FFCI: A Framework for Interpretable Automatic Evaluation of Summarization

In this paper, we propose FFCI, a framework for fine-grained summarization evaluation that comprises four elements: faithfulness (degree of factual consistency with the source), focus (precision summary content relative to reference), coverage (recall and inter-sentential coherence (document fluency between adjacent sentences). We construct novel dataset focus, coverage, coherence, develop automatic methods evaluating each dimensions FFCI based on cross-comparison metrics model-based methods, including question answering (QA) approaches, semantic textual similarity (STS), next-sentence prediction (NSP), scores derived from 19 pre-trained language models. then apply developed in broad range models across two datasets, some surprising findings.